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Abstract: When service robots carry on mission planning and execution, they not only need structural information about 
the environment to navigation and localization, but also need to possess deeper knowledge to endow a robot with higher 
degrees of autonomy and intelligence. However it is hard for a robot to cognize the semantic description such as the func- 
tion attributes of objects and rooms only relying on robot’s sensors itself. When the exploration scope of the robot is ex- 
panded to large regions such as a building, the problem becomes more prominent. In this paper, artificial labels are used to 
provide semantic concepts and relations of objects and rooms in a building, which can solve the complexity and limita- 
tions of semantic recognition and scene understanding only with robot’s vision. Imitating spatial cognizing mechanism of 
human, three hierarchy space model is built for a building service robot. With this space model, not only the plane struc- 
ture of the environment for navigation is known, but also the abundant semantic information in different kinds of envi- 
ronment are obtained. Simulation results show that the three hierarchy indoor space model is applicable to service tasks of 
indoor robot, understanding human semantic statement, produce reasonable service path, and ensuring the security of ro- 


bot running in building environment. 
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1. INTRODUCTION 


The emergences of service robots in our work and life 
bring us lots of conveniences more than ever before. To 
perform these tasks, the robot should move freely in varied 
scenes of a building such as corridors, elevator, living room 
and printing room. Thereby, the moving scope of the robot 
becomes larger and larger. Aiming to large scale unknown 
environment, the study of SLAM has become the hotspot of 
current research for map building. Many in-depth researches 
have been done to solve data association, path exploring, 
identification of static and moving objects in dynamic 
environment, and understanding & extraction of the 
environmental characteristics, etc. However, due to the 
restriction of the robot’s perceptual ability, some key 
problems are still unsolved that restrict map’s application to 
robot navigation [1-5]. 


Semantic map is illumed by human’s task performing 
steps. Semantic map should include detailed information 
about the function property of rooms and objects [6, 7]. The 
locations of objects can show the semantic functions. For 
example, there must have printer in printing room. Thus 
using the location information of objects, semantic path 
planning and navigation can be realized. Identifying large 
objects and doors are important to build semantic maps. 


High-level semantic feature extraction algorithm is used 
to identify the objects and the doors, which are regarded as a 
partition warranty to form a global topological map, and the 
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object relationships in the room are used to form object asso- 
ciated map which is treated as a local map. However, the 
morphological changes of the door, different visual angle, 
illumination change, shelter and other matters will affect the 
accuracy of recognition [8, 9]. The problems mentioned 
above result in unsatisfactory identified result, and functional 
properties of objects are difficult to be obtained relying 
solely on robot vision [8-10]. Illumination change, viewing 
angles, shelter and other issues also affect its recognition 
accuracy. And the operation method, function and other 
properties of the objects are impossible to be obtained only 
by visual observations. As the moving scope of the robot is 
becoming larger and larger, such as a building, the process of 
semantic map building is more and more difficult. Combining 
with layered map building idea, using QR code based artificial 
labels, three hierarchy space model is a significative attempt 
to easily build semantic map for a building service robot. 


2. THREE HIERARCHY SPACE MODEL AND THE 
DESIGN OF ARTIFICIAL LABELS 


2.1. Three Hierarchy Space Model Applying to Building 
Service 


In order to make a service robot be intelligent, we should 
get inspired from the representation method of the space 
environment used by people, and study the spatial represen- 
tation of service robots to relate with human language infor- 
mation. Human’s understanding to the environment is car- 
ried out at different levels: the concepts of storey and corri- 
dors are constructed for the large space; apartments and 
rooms are formatted for the small space; the objects operated 
by robots are located at somewhere of a room, which is 
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Fig. (1). The structure of three hierarchy space model. 


looked as local smaller area. The corners and doorplates are 
key features which use as navigation basis to the people; the 
placement of large items in unit room and the relations be- 
tween objects and rooms can provide human the basis of 
region path planning; the searching and grasping of the ob- 
jects in local complex space can be based on three- 
dimensional space model around the object receiving from 
human’s eyes. Human uses three level cognizing methods to 
remember the environment from macroscopic to microscopic 
view, from broad categorizing style to the details. 


Therefore, the robot can imitate spatial modeling method 
of the human and build indoor three hierarchy environment 
maps in unknown environment. During the map building, the 
semantic information is added to describe the function prop- 
erty and ascription relation of objects, rooms, corridors and 
floors. According to the service features of the building, the 
three hierarchy space model is built which is shown as Fig. (1). 


2.2. The design of artificial signpost label and flat label 


QR code (Quick Response code) technology coming 
from Denso Corporation of Japan is the most representative 
two- dimensional bar code currently. In addition to the ad- 
vantage of large capacity, high reliability, strong security and 
confidentiality, its recognition orientation can reach 360 de- 
grees, and it is effective in representation of character [10, 
11]. 


The semantic describing information of the environment 
is transformed to data encoding, then QR code is formed. QR 
code based labels are mainly pasted at three kinds of 
positions in three-level indoor map model: the key positions 
in corridor area; the doors of flatlet; the large objects in 
rooms. The first is called artificial signpost label, which 
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mainly provides the semantic information of navigation and 
context description in large scale environment; the second is 
called artificial flat label, which mainly provides the struc- 
ture information of the suites; the third is called artificial 
object label, which mainly provides the information of large 
objects such as name, shape, function and attribute. 


Signpost label and flat label are all used in corridor. Cor- 
ridor is a typical structured environment that has single hues, 
dominated by cold hues. To identify the artificial label 
quickly from long distance and estimate the relative location 
of the robot and the artificial label, color and graph based 
artificial signpost label and flat label are designed, shown as 
Figs. (2a and 2b). 





(b) flat label (c) object label 


(a) signpost label 
Fig. (2). QR code based artificial labels. 


The label is made up of two parts: periphery mode and 
the inside information. The artificial object label's periphery 
is named as mode part which includes colored rings and blue 
rectangular; and the insider is named as information part 
which includes QR code. The coding of signpost label 
mainly stores the semantic info of current location, area 
structure info, the function semantic info and context info for 
navigation. The coding of flat label mainly stores the struc- 
ture info of the room. The periphery modes of signpost and 
flat label are similar, made up of one color circle and a blue 
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box inside. The color of the circle is decided by the pasted 
location of the label. The structure in the flatlet is more com- 
plex than that in corridor. There are many kinds of objects, 
and the artificial labels pasted on large objects are often 
shaded. The detailed realization method of the artificial label 
recognition and closing refers to the thesis [11]. 


3. THREE HIERARCHY MAP BUILDING STRATEGY 
BASED ON THE FEATURE OF DIFFERENT ENVI- 
RONMENT 


3.1. Global Semantic Topology Map Building based on 
Distributed Information Representation 


Floors and corridors are structuring environment with a 
spot of typical features, so building topology map including 
nodes coordinate information is traditional mapping method. 
It can reflect the positions and relations of nodes. However, 
when the physical position of the node is uncertain, it 1s dif- 
ficult to achieve the robot’s exact navigation, which must 
reduce robot autonomy. According to the signpost navigation 
patterns of the human, the artificial signpost labels based on 
QR code technology are distributed in the key positions of 
the environment to form cognitive-oriented points. It can not 
only give semantic information of navigation, but also pro- 
vide the semantic characteristic of the environment with re- 
spect to mapping. Based on the fusing localization technol- 
ogy of intelligent space, the global fuzzy topology semantic 
map with “distributed information representation” is formed. 
The map has three specialties: global, semantic, and topo- 
logical, which are showed in Figs. (1e and If). 


At the cognitive-oriented point in the semi-unknown en- 
vironment, the robot decides its moving direction based on 
the artificial labels and realizes navigation. At the same time, 
the robot builds a time-space network which can reflect time 
and space relationship between nodes. Consider the robot 
itself as a centre to establish a topological neighborhood 
around the current node position. That likes the biological 
nervous system in which excited neuron makes the surround- 
ing neurons excite together. The relation weight between the 
current node and the neighborhood node will update with it. 
Immune network algorithmis used to form the environmental 
awareness mechanism of "distributed representation", then 
the global semantic topology map is built. 


3.1.1. Time-Sequential Memory Mechanism of Network 
Nodes 


Immune cells in the network have self-attenuation. That 
is, if the external input doesn’t stimulate it, the cell’s activi- 
ties will gradually decrease. The vitality of immune cell is 
defined as below: 


0 Va, (t)=1 
a (=; 1 f(t) =1&al (t-1) =0. (1) 
na, (t-1) else 


f.(t)=1 means the scene tag of the node i at time f is 


found. The vitality of antibody reflects the distance between 
nodes. 
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3.1.2. Growth Mechanism of Immune Cells 


The emergence of each new sequence will produce a new 
immune cell and activate the immune cells. At the same time 
a connection with an existing active cell is brought. The 
connection weight w; will reflect when the foregoing anti- 
body j give maximum impact on cell i. 


0 t=0 
O,(t)=\ka(Ha(t-l) a (t)=0&a,(t)=1. 
w, (¢—1) else 


(2) 

a;(t) is the activity of foregoing antibody j(output), a;(t) is 
the activity of the antibody i. k is learning regulation rate. If 
the output of the foregoing antibody is larger, antibody activ- 
ity will be stronger, and the connection weights will be 
stronger too. 


3.2. Regional Semantic Planning Map Building based on 
QR Code Technology 


When a robot moves into a suite, it will build 2D hybrid 
map of the suite. As the robot works in the room, it will find 
some artificial object labels pasted on large objects. 


The name, shape, color, size, functionality and other at- 
tribute information of large objects have been written in QR 
code. The robot reads QR code in the observed image, and 
adds the information into the object function database imme- 
diately. Simultaneously, the correlation between observation 
position (node) and the corresponding object is established, 
the relations of objects and rooms are formulated, which are 
supplemented to object relation database. Object function 
database and relation database are combined to object infor- 
mation database, which is the fundamental guarantee for 
management of household goods. 


When the regional topology map based on flat labels is 
established, the nodes(the positions where the robot observes 
the objects.) in the same room will form a set that corre- 
sponds to the concept of room. The four blue clouds in Fig. 
(3) represent four certain room areas. Using the established 
attributive relation between observation positions (nodes) 
and object, the object can be attributable to the room associ- 
ated with the nodes. That is, as the function of items is 
known, the room’s function can be determined according to 
the object’s function. Thereby the robot can know the at- 
tributive relation between rooms and objects. Shown as Fig. 
(3), the robot recognizes the bookcase which belongs to 
room 2”. Thinking of the bookcase’s function, the robot can 
conclude that the function of room 2” is a study. By the same 
token, room de: 3" and 4° are named as kitchen, bedroom and 
living room respectively. Then function semantic topology 
map is got which not only contains topology relation be- 
tween rooms, but also includes semantic description of ob- 
ject function, objects adscription and relation, and room 
functions. 


The set of object information database is: 
Sp =thatassl.5 fo In that, 
[,=Ww@gOw|veV,geG,,weW}. (3) 
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Fig. (3). The diagrammatic sketch of a regional semantic map. 
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Fig. (4). The relation of the robot and observation position. 


I, is the set of object information, V is the set of obser- 
vation nodes in the environment, G,, is the set of object at- 


tributes, W is the set of divided rooms; v is the specific node 
corresponding to the object, g is the specific property of the 
object, w is the attributive room where the object exists, " ® " 
represents correlation. 


3.3. The Local 3D Grid Map Building based on Occupy 
Probability Model 


As the appearance of large numbers of humanoid robots 
and more complex indoor structure, the simple 2D informa- 
tion can’t give general description for the surrounding. If the 
robot imitates human’s object searching mode in local area 
to build 3D map for complicated environment, as shown in 
Fig. (la), the robot’s task completion can be satisfied. 


Using binocular vision schlepped on the robot, the depth 
information of the local small space are got by SIFT algo- 
rithm, and the uncertain mathematic models of binocular 
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vision information are structured based on DSmT evidence 
theory. 


The indoor environment is defined as 3-D grid space 


which set is K. Each grid cell is called as voxel Caye K IS 
the set of all voxels. 


The definition of Voxel: in the true 3D space of geogra- 
phy information system, it is the smallest volume cell with 
fixed size which is used to describe the data memory, man- 
agement and expression. Voxel is a cube with side length ò. 


oe(c, 7 „) is defined as O&E value which describes the 


coverage degree of each voxel with the value range as [0, 1]. 
The value of oe(c, ; .) means three kinds of coverage con- 


dition: occupied, empty and unknown. 


Suppose the robot is located at the dot ‘a’ currently and 
the coordinate is Q,, which is shown as Fig. (4). The position 
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(a) 


Fig. (5). Leader 1-DX robot. 


of the matching feature point o; is described as b in Fig. (4) 
which is observed at a. The line linking from a to b forms the 
vector 5,,¢€ P’. The O&E probability values of the voxels 


drilled through by the vector Sag will be changed during 
this observation. The set of the voxels which are drilled 
through can be defined as formation (4): 


Z(Sab,4) = {co liE N,i Sn}. (4) 


In formation (5), 7 means the numbers of the voxels be- 
ing drilled through. Q; means the coordinate of the i th voxel 
which is drilled through. 


> 











0, =0, A ET (5) 
Sab 
n = in eel +1. (6) 


Thinking of the voxel b’ behind the obstacle position b, 
it should be confirmed as occupying state, so the formation 
(7) is got. 


= 


[se 


= 


eho (7) 


Sa,b | 


Then the set of voxels, effected from the robot position a 
to the feature point b, is defined as below. 


Z, =L(Sav,a)U{B}. (8) 


Owing to the influence of unknown environmental 
factors and the limitations of robot body sensors, it is hard to 
eliminate the uncertainty of the perceptive information. 
Therefore, when the robot perceives the environment each 
time, in addition to modeling for the uncertain information, 
we need to fuse the information perceived by the robot. 
DSmT algorithm can be used to fuse the imprecise and high 
contradictory information statically or dynamically. 








Q, = Q, + (int( 











The reliability, the decision-making rules and 
combination rules in DSmT are based on super-power set 
space. Set two focal elements 9, and @, in the identification 


box@. The circumscription 6, means that voxel is occupied 


and @, means that voxel is empty. Its super-power set space 
is D° ={0,6,,0,,0,(10,,0,U0,}. @ means that this 
information fusion is invalid. m(0,) means the occupied 
reliability evaluating function. m(@,) means the vacant 
reliability evaluating function. m(@,{\@,) is the conflict 


factor’s reliability evaluating function, which means the con- 
flicting situation of occupy or empty. m(@,U6@,) means the 


uncertain reliability evaluating function. 


The proportional conflict redistribution rule (PCR) is 
adopted to fuse the basic reliability of many information got 
by info source, and the syncretic reliability calculation can 
be simplified as: 


Mcp (O,) = m (9, )m,(8,) + m (0, )m, (0, U8, )+m (0, UO, )m,(0,) 


4 m,(0,) m, (0,) + m,(6,)° m (0,) (9) 
m,(0,)+m,(0,) m,(0,)+m,(0,) 


M pcr (0, ) = m (0, )m, (80, ) + m, (8, )m, (0, U0, )+m (0, U0, )m,(0,) 


mM, (0,)} m, (0, ) + m, (0,)°m, (0, ) (10) 
m,(0,)+m,(8,) m,(0,)+m,(0,) | 


4. THE EXPERIMENT RESULT AND ANALYSE 


Leader 1-DX mobile robot platform (shown as Fig. 5) is 
developed by our lab, which equips laser sensor and Point 
Grey Research's Bumblebee®2 Stereo vision camera sys- 
tems. Laser sensor is used to avoid barriers. Vision camera 
systems can collect indoor environmental information abun- 
dantly. Fig. (5b) is a typical building in our school which has 
three floors. Several regions are divided in every storey, and 
three kinds of artificial labels are pasted on selected posi- 
tions. Three kinds of experiments have been done in corri- 
dors, suites and some typical local areas. 


4.1. The Building Experiment of Global Semantic Topol- 
ogy Map in the Corridor 


The robot moves in a corridor shown as Fig. (5b). The 
start is stairway ostium, the corresponding node number is 1. 
Based on the leading of artificial signpost labels, the robot 
goes through 2~6 nodes in order, finally back to the starting 
node. The No. 1 node has been pastedl-type signpost label, 
No. 2,4,5 nodes have been pasted lI -type signpost labels. 
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Fig. (6). The mapping process in the corridor of the building. 


No. 3 and 6 nodes have been pasted flat labels. The data 
written in the three kinds of labels are shown in Fig. (6). 


4.2. The Building Experiment of Regional Semantic Plan- 
ning Map 


The experiment is done in a quasi structured simulant 
home environmental shown in Fig. (7a). Artificial object 
labels are pasted on TV cabinets, sink, stove, beds and other 
items. 








(b) regional semantic map 


Fig. (7). Global semantic planning map building. 


Image data is real-time collected by the robot. Figs. (8a 
and 8b) are two images collected at different observation 


points. The label pattern collected from image based on the 
method of artificial label recognition is shown in Figs. (8c 
and 8d). QR code is read from (c) and (d). As the QR code 
information in (c) and (d) are both "TV cabinet," the two 
results are consistent. 





(c) (d) 
Fig. (8). Identification and reading of the artificial object label. 


The robot moves as the velocity of 0.3m/s. The robot 
starts at arbitrary position and explores the surrounding. A 
lot of dots shown in Fig. (7a) represent all observation points 
got by the robot. 


Undirected weight graph is built in the end of explora- 
tion, as the dotted lines shown in Fig. (7b). The nodes in the 
undirected weight graph are the observed positions stayed by 
the robot. The relative graph shows the passing paths for the 
robot when it does service task. Simultaneously the attribu- 
tive relations between objects and nodes are obtained (black 
two-way arrows shown in Fig. (7b)). Four separate sets (the 
four thick solid ellipses in Fig. (7b)) represent four rooms. 
The global topological map is built which is partly described 
by the thick solid green line in Fig. (7b). According to the 
attributive relations between nodes and objects, we can de- 
termine the attributive relations between objects and room, 
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Fig. (10) The matching result and map building result in two scenes. 


and then determine the functional properties of the room. Up 
to now, the robot obtains a series of results which include 
local topology graph (undirected weight graph), regional 
semantic planning map with room segmentation, the topol- 
ogy relations of nodes, attributive relations between objects, 
nodes and rooms, the functional property of room, and so on. 


4.3. The Building Experiment of Local Space 3D Map 
Suppose n=[0.1,0.9,0] at time ¢, that means the un- 


known belief value is 0. Suppose the unknown value at time 
t+1 is 0. The occupying belief value increases and the empty 
value decreases, but their sum is 1. Then 10 cases come into 
being shown as below which form the matrix m. 
0.1 0.2 03 04 0.5 06 0.7 0.8 0.9 1 
m =|0.9 0.8 0.7 06 0.5 04 03 0.2 0.1 0 
0 0 0 0 0 0 0 0 0 +0 


The fusion result of the four kinds of fusion rules are 
shown as Fig. (9). 


As shown in Fig. (9), while abnegating process for all 
conflict, the fusion result of XQ rule deviates from the basic 


fusion law. The higher the conflict, the greater the bias 
value, or even wrong fusion result. The DSM rule is better 
than XQ rule, which is close to the fusion law as less conflict 
and lapses from the fusion law as biggish conflict. The fus- 
ing outcomes of DS and PCR rules essentially accord with 
the fusion law. When the conflict is small, the effect of the 
two fusion algorithm is the same. As the conflict higher, the 
PCR rule more accords with the fusion law, however DS rule 
has a little windage. As the conflict reaches to the highest 
value, DS rule’s fuses result is wrong. 


Fig. (10) showed the building results of 3D description 
map. When the door is closed, doorsill and door are in the 
same plane. However, the deep information at the doorsill is 
different from other features around it when the door is 
opened. Using this 3D description map, the doorsill is easy 
to be detected and the robot can use corresponding strategy 
to get across the door. 


CONCLUSION 


According to the service features of the building robot, 
imitating the ambient cognitive model of the human, three 
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hierarchy space model is found for building environment 
based on the localization of multi-information fusion in intel- 
ligent space. Artificial signpost labels are pasted in wide area 
environment such as corridors, which give navigation infor- 
mation for robots. Artificial object labels are pasted on large 
objects in the flatlet, which give the information of objects 
such as their function, property and ascription relation. The 
simulation and experiment results in three kinds of typical 
environments showed the effectiveness of the algorithm. 
Some problems should be studied deeply, that is, the real- 
time of the algorithm and the complication of the space 
should be thought for building three hierarchy space model. 
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